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ABSTRACT 

A procedure used to average several Mantel-Haenszel 
delta difference values for an item is descrilDed and evaluated. The 
differential item functioning (DIF) procedure used by the Educational 
Testing Service (ETS) is based on the Mantel-Haenszel statistical 
technique for studying matched groups. It is standard procedure at 
ETS to analyze test items for their differential difficulty with 
regard to gender and race/ethnicity of examinees. Test items used to 
perform the averaging procedure for this study were taken from the 
item bank of a large national testing program. Each form used in this 
study consisted of 80 items. Forms selected for the study had a large 
number of items in common. The delta difference average is the mean 
of single delta difference statistics across forms weighted inversely 
to their respective error variances. Overall, the results of 
differential item functioning analyses are similar to the weighted 
average procedure. Even if test forms were constructed to have items 
in common, a55 is the case in common item equating, the cost of test 
development and subsequent statistical analyses would be 
considerable. The weighted average procedure, on the other hand, is 
quite easy to perform, particularly with the use of a personal 
computer, and it does not require reformatting of the original data. 
Seven data tables are included. (TJH) 
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It is A standard procedure at ETS to analyze test items for their 
differential difficulty with regard to gender and race/ethnicity 
of test takers. The differential item functioning (DIF) procedure 
(Holland & Thayer, 1986) used at ETS is based on the Mantel-Kaenszel 
statistical technJQue (Mantel & Haenszel, 1959) for studying matched 
groups. The Mantel -Haenszel method compares the odds of two gender 
or racial/ethnic groups answering a test item correctly when the 
members have been matched on the basis of test scores. The DIF 
indices aid in identifying differences in difficulty caused by 
characteristics of the test item itself, after real differences 
in pertinent knowledge and skills have been taken into account by the 
process of matching. 

Groups which are identified for the purpose of DIF analysis are 
referred to as* reference ( generally males and Whites) and focal 
(generally females. Blacks. Asians. Hispanics. etc.). Members of 
the reference and focal groups are divided into subgroups based 
on the total i:?st score. Then, the odds that a reference group 
meJuer answers th»5 item correctly are calculated for every matched 
subgroup. This calculation is repeated for members of every matched 
focal group. The next step is to compute the ratio of the ouds 
for the focal group to the odds for the reference group for each 
matched subgroup. Then, these odds ratios are averaged across 
the entire score scale, weighted according to the number of 
individuals at each score value. The estimate resulting from this 
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procedure is the average factor by which the odds that a reference 
group member answers the item correctly exceeds the corresponding 
odds for comparable focal group members. For example, if the 
odds-ratio value i» equal to 1, then both subgroups are equally 
likely to answer the item correctly or, to state it differently, 
the item is equally difficult for both subgroups. The MH D-DIF 
(Mantel -Haenszel delta difference) index, commonly used at ETS, 
is a scaled odds-ratio. The scale is the ETS delta scale used i;. 
test construction and analysis. Delta is a transformation of percent 
correct to a scale for item difficulty ranging to about 6 for very 
easy items to about 20 for very difficult items. A delta value of 13 
corresponds to 50 percent correc;,. Consequently, if MH D-DIF is equal 
to -1.00, it means that the iteir is one delta unit more difficult for 
the focal group than it is tor comparable members of the reference 
group. Near the middle of the difficulty scali^, a difference of 
1 delta point is equivalent to about 10 percent difference in 
percent correct. The interpretation would be reversed for positive 
values of MH D-DIF. 

Based on the absolute value of MH D-DIF statistics and their level of 
significance («-.05), test items are classified into three categories. 
The three categories are labeled A, B, and C. Category A contains 
items with MH D-DIF between -1 and +1 or MH D-DIF not significantly 
different from 0. Category B is assigned to items with MH D-^^F 
significantly different from 0, and an absolute value of at least 1, 
but not significantly greater than 1. Category C characterizes itecs 
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with absolute value of MH D-DIF at least equal to 1.5, and 
significantly greater than 1. 

Each ETS testing program that meets minimian sample size requirements 
petforms DIF analyses. This paper refers to DIF analyses completed 
ordinarily for test assembly purposes, with sample size requirements 
for the focal and reference groups combined of 500 or greater, and for 
the focal group of 100 or greater. Some testing programs administer 
new forms more than once during the same 12-month testing cycle. 
As a result, items that appear in multiple forms are analyzed several 
times and have multiple MH D-DIF statistics and classifications. 
Since each MH D-DIF is influenced by the difficulty of other items 
included in any particular form, the magnitude, significance, and 
classification of each MH D-DIF differs occasionally. Consequently, 
test developers have to decide which MH D-DIF value to use during the 
test assembly process. Currently, the choices involve using the first 
value, the most recent value, the largest (worst case) value, or some 
average value (such as the median). 

t 

The purpose of this paper is to describe a procedure suggested by 
Paul Holland (personal communication. May 16, 1989) that can be used 
to average several MH D-DIF values for an item, to test the 
significance of the average, and to obtain a new classification. 
This procedure is not exact since each MH D-DIF value for the same 
item was calculated with different sets of items comprising each test 
form and, consequently, the criterion score. Therefore, this 
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averaging is compared with the exact technique that involves combining 
aami^les to whom the common items were administered and recalculating 
MH D-DIF statistics from item response data. 



Instrum>>r.ffi 

Test items used to perform the zveragin^ procedure were taken from 
the item bank of a large national testing program. Each form used in 
this study consisted of 80 items. Three pairs of forms that contair^d 
overlapping items were selected. Form 7 and Form 11 had 30 common 
items, Form 4 and For= 6 had 48 items in common, and Form 3 and Form 5 
had 52 items in common. These forms were selected for this study 
because they have a large number of items in common. Ordinarily, 
a smaller number of items overlaps between forms. Forms with large 
number of common items were selected for this study in order to 
provide a meaningful matching criterion score, i.e. in addition to 
the total test score criterion, the common items score criterion was 
also used in DIF analyses. 

Method 

The MH D-DIF average is the mean of single MH D-DIF statistics across 
forms weighted inversly to their respective error variances. 
The first step therefore is to calculate weights proportional to the 
inverse of error variances with the constraint that the sum .of the 
weihgts must be equal to 1, w^-klj^ andlw^-l. The second step is to 
compute the MH D-DIF weighted mean, MHAVG-ZMH D-DIF^w^. In the third 
step, the standard error of the weighted average is calculated by 



taking a square root of the aum of the squared weights (from step 1) 
multiplied by the original error variances, SEW-/zw^SE^ . The last 
step is to perform a significance test for the average MH D-DIF using 
the new standard error and assign a category (A, B, or C) depending 
on the outcome. The statistical significance was tested using z-test 
Mid a -.05. All computations were performed with thp use of 
SPSS-X/PC+. 



For the second part of this study, standard DIF analyses were 
performed. Each test form was analyzed twice, first with the total 
test criterion score, and next with the common items criterion score. 
These analyses were performed using single samples of examinees who 
took either of the paired forms. In the next step, DIF analyses were 
executed using combined samples of canaidates to whom each pair of 
test forms was administered, i.e. examinees who took Form 7 and 
those who took Form 11 were grouped togeth- r for each analysis. Two 
DIF analyses -were run for each pair of forms, one with the total test 
score as the criterion and second with the common items score as the 
criterion. Altogether, six analyses were performed for each pair 
of forms. The following example lists analyses for Forms 7 and 11: 

- Form 7, single group, total test score criterion 

- Form 7, single group, common items score criterion 

- Form 11, single group, total test score criterion 

- Form 11, single group, common items score criterion 

- Form 7 & 11, combined groups, total test score criterion 

• Form 7 & 11, combined groups, conmion items score criterion. 
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Table 1 presents sample sizes for male/female comparisons for each 
test form used In this study. Table 2 and Table 3 present results 
for Forms 7 and 11, Tables 4 and Table 5 present information for 
Forms 4 and 6, and Table 6 and Table 7 contain results of analyses 
for Forms 3 and 5. Each table lists the item number in each form, 
MH D-DIF values, standard errors, and classifications obtained 
from separate analyses with single sample sizes. In addition, each 
table contains MH D-DIF statistics, s'candard errors, and 
classifications obtained from analyses performed with combined 
samples. Finally, each table contains results of the weighted average 
procedure. The mean values of MH D-DIF statistics and standard errors 
are listed at the bottom of each table. 

Comparisons between the exact procedure which involves combining 
samples and recalculating CIF statistics, and the weighted average 
procedure, should be the most straightforward when the common item 
criterion score is used. In this case, the influence or confounding 
caused by the*"uncommon" items, that is items different in each form 
which also contributi;d to the total test score criterion, is 
eliminated. The following description is focused on the results of 
analyses with combined samples and on the results of the averaging. 
The results of single form analyses are listed for reference purposes. 

When the criterion score of 3f common items was used to analyze 
Forms 7 and 11 using combined samples, DIF analyses resulted in two 
C classifications, for items 3 and 95, and five B classifications, 
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for items 33, 38, 42, 68, 85. The remaining items were classified as 
A's. The weighted average procedure that was performed on MH D-DIF 
values listed in Table 2, resulted in identical classifications for 
the same items. Table 3 presents the outcomes of analyses with the 
total test score as the criterion for Forms 7 and 11. Although the 
comparisons are not as straightforward, this situation is more like 
that in which the procedure would be applied in a typical testing 
situation. The combined samples analysis resulted in one C 
classification, for item 95, and seven B classification;?, for items 
3. 16, 38, 42, 68, 75, 85. The weighted average procedure resulted 
ill one C classification, for item 95, and five B classifications, 
for it«ms 16, 38, 42, 75, 85. The two procedures, combined samples 
and weighted average with total score as the criterion, differed by 
one classifications for item 68. This item was classified as B with 
combined samples, but when the averaging procedure was used, it was 
classified as^A. Item 68 obtained MH D-DIF of -1.07 with the combined 
samples appi^oach, and - .97 with the weighted average method. The two 
values differed only by .10, but since one of the conditions used for 
classifying items as B is that the absolute value of their MH D-DIF 
must be at least 1, MH D-DIF equal to -1.0? was classified as B, 
whereas MH D-DIF equal to -.97 obtained A. 

Tables 4 and 5 contain information about Form 4 and Form 6. These 
two forms had 48 items in common. Table 4 presents results of the 
analyses with the 48 items score as the criterion. The combined 
samples and the weighted average procedures each resulted in ten 
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B classifications, for items 32, 38, 44, 45, 46, 70, 71, 72, 74, 76. 
Table 5 contains results of DIF analyses with the total test score 
as the criterion. The combined samples procedure and the averaging 
procedure each resulted in eight B classifications, for items 
32, 38, 44, 45, 46, 71, 74, 76. Items 70 and 72 were classified as A 
with the total test score criterion, but as B with the commom items 
criterion. This difference in classification could be explained by 
the influence of the unique items that were also included in the total 
test score criterion. 

Tables 6 and 7 present results of DIF analyses for Form 3 and Form 5 
with 52 common items. When the common items score criterion was 
used (Table 6), items 45, 52, 73, 75, 93 were classified as B by both, 
the combined samples method and the weighted average method. 
The difference in classification, for items 35 and 64, was due to 
the convention used for assigning the B category. The value of 
MH D-DIF must not only be significantly different from 0 but irs 
absolute value must be at least equal to 3 , Item 80 obtained MH D DIF 
of -1.24 with combined s^ples which was equal to the critical value 
at a-. 05. Therefore, the result of the significance test yielded 
an A, since the tssted value must exceed the critical value for 
the to be rejected. When the total test score was used as the 
criterion (Table 7), items 45, 52, 60, 73, 75, 93 were categorized 
as B by both, the combined samples method and weighted average method. 
Item 64 obtained borderline MH D-DIF values of -1.11 with combined 
samples and -.99 when the weighted average was used. 
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Educatlorml Imporfc^ncg nf Si^udy 

The technique of averaging MH D-DIF statistics presented in this paper 
makes use of multiple outcomes of DIP analyses performed on the same 
items. When multiple administrations of the same items in different 
test forms occur, the best procedure is to combine all samples to 
which these items were administered and perform new DIP analysis. 
This procedure, however, requires additional reformatting of the data 
files and, therefore, is inefficient and costly. Also, there are 
seldom enough common Items to provide a meaningful matching criterion. 
If these ••combined'' results are not available, test developers must 
choose among multiple MH D-DIP statistics, t^ere the most recent 
MH D-DIP may be selected as opposed to the ••older^^ ones, or the 
largest or median value may be used to Judge differential 
performance of items. If the largest or •'worst- case •• value is 
systematically selected, number of B and C items will increase 
as the number of analyses increases. Although the averaging procedure 
makes use of all MH D-DIP statistics and their standard errors, it is 
not an exact procedure because the matching criterion, that of total 
ttest score, is diffferert each time MH D-DIF is computed (not all items 
are identical between forms) . 

Overall, the results of DIP analyses, with combined samples and the 
common items score as the criterion and the total test score as the 
criterion, are similar to the weighted average procedure when each 
of the above criteria was used. However, no statistical test was 
used to decide whether the difference between each pair of absolute 
MH D-DIP values was significant. Prom the practical significance 
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standpoint, classifications resulting from the averaging procedure 
were similar enough to classifications obtained from combining 
samples and recalculating MH D-DIF statistics. In one case (Forms 
4 and 6). both procedures resulted In Identical classifications for 
the same Items. DIE analysis with the total test criterion score 
resulted In one different classlflc(»tlon for Forms 7 and 11. When 
Forms 3 and 5 were analyzed with the common Items criterion, three 
Items were classify •d differently by the combined samples method and 
the weighted average approach. With the total test score criterion, 
Forms 3 and 5 differed by one classification when these two methods 
were used. The magnitude of differences between the absolute values 
of m D-DIF and their standard errors for the discrepant cases was 
very small. 



Even If test forms were constructed to have Items in common, 
as is the case in common item equating, the cost of test development 
and subsequent statistical analyses would be considerable. Also, 
when the total test score is used as the criterion with combined 
samples, it is not possible, at the present time, to refine the 
criterion by deleting C items. This problem involves using the 
total test score for each examinee, as opposed to each person's 
single responses to each test item. The weighted average procedure, 
on the other hand, is quite easy to perform especially with the use 
of a PC, and it does not require reformatting of the original data. 
However, if the weighted average technique were to be used 
operationally in other testing programs or for other focal groups, 
further analyses should be performed with the other reference and 
focal groups. 
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TABLE 1 

Stunple Sizes for Reference «nd Focal Groups 





FORM 3 


FORM 4 


FORM 5 


FORM 6 


FORM 7 


FORM 11 


MALE 


395 


299 


346 


340 


378 


277 


FEMALE 


332 


240 


274 


258 


316 


259 


TOTAL 


727 


539 


620 


598 


694 


536 
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TAILE 2 

Singlt, Cowbintd, and Avtrigtd OIF StvtUtict 
Crittrlon • Comon itMM Scort 



ITEM 


CUSS* NK-DOIP 


SE 




CLASS 


iv*iiui r 


s 


C 


•1.90 


0.51 


43 




1 .Or 


4 


8 


-1.16 


0.52 


39 




- u.oo 


S 




-0.21 


0.43 


54 




0 HI 


7 




•0.36 


0.45 


64 




U. . I 


9 




-0.65 


0.57 






• n oo 


10 




0.45 


0.41 


82 




A M 

U.09 


16 




0.73 


0.43 


1 

1 




1 .uu 


18 




-0.45 


0.99 






1 


26 




-1.72 


1.49 


72 

r K 






28 




-0.03 


0.38 


117 
Or 




n en 


33 


K 


-1.21 


0.53 






- 1 .33 


37 




0.33 


0.38 






n 


38 


8 


1.59 


0.57 






1 Tn 

1 .rU 


42 


8 


1.90 


0.78 


27 




U.Oc 


U 




0.41 


0.51 






-u.2o 


53 




0.55 


0.41 


oo 






60 




-0.63 


0.56 


28 




-1.02 


62 




-0.26 


0.40 


90 




-0.59 


67 




'-0.97 


0.58 


31 






68 


8 


•1.34 


6.41 


89 


S 


-1.11 


75 




0.93 . 


0.41 


91 




0.B8 


81 




•0.63 


0.49 


38 




-0.24 


83 




•0.49 


0.67 


67 




•0.18 


85 


C 


2.47 


C.71 


29 




-0.37 


86 




•0.55 


0.38 


94 




•0.83 


S8 




0.22 


0.67 


22 




-0.28 


69 




•0.06 


0.44 


63 




0.77 


91 


8 


1.12 


0.47 


61 




0.64 


92 




0.74 


0.44 


73 




0.72 


95 


C 


2.10 


0.56 


41 


1 


1.66 


MEAN 




-0.42 


0.55 






•0.47 



*ll«nk tpactt indlcitt A \tmm 



C0N8INED AVERAGE 







mn D-DIr 




CLASS 


MK D-DIF 


SE 


n AA 


c 


- 1 .rO 


A VA 

0.39 


C 


-1.80 


0.39 


n u 




-D.Of 


A VA 

0.39 




-0.92 


0.39 


0 LA 




U.cr 


A V9 




0*27 


0.32 






•U.^f 


A 

U.39 




-0.52 


0.35 


u.o% 




*U.04 


A ^1 

0.42 




-0*77 


0.43 


0 A7 




M.52 


0.31 




0.54 


0.31 






0.83 


0.32 




0.85 


0.33 


n oi 

u.y 1 




•0.96 


0.64 




•1.01 


0.67 


1 •no 




•1.55 


1.02 




-1.63 


1.04 






0.15 


0.28 




0.22 


0.29 


n An 

U.OU 


■ 


•1.19 


0.30 


1 


•1.27 


0.40 


n zz 
u.^ 




0.30 


0.29 




0.34 


0.29 


n At 


■ 


1.54 


0.42 


B 


1.64 


0.42 


U. fO 


■ 


1 .27 


0.54 


8 


1.36 


0.55 


U.D1 




A 4 A 

n.iu 


0.31 




0.10 


0.39 


n Z.A 
u.^ 




0.13 


0.30 




0.12 


0.31 


0.67 




•0.81 


0.43 




•0.81 


0.43 


0.46 




•0.45 


0.30 




•0.41 


0.30 


n AA 
u.oo 




A 

•0.76 


0.43 




•0.76 


0.44 


0.46 


1 


•1.32 


0.30 


8 


•1.23 


0.31 


0.46 




0.94 


0.30 




0.91 


0.31 


0.60 




•0.47 


C.38 




•0.45 


0.38 


0.86 




•0.28 


0.52 




•0.35 


0.53 


0.67 


1 


1.07 


0.47 


1 


1.01 


0.49 


0.44 




•0.71 


0.29 




•0.6f 


0.29 


0.80 




•0.14 


0.51 




•0.01 


0.52 


0.50 




0.26 


0.33 




0.33 


0.33 


0.48 




0.76 


0.34 




0.86 


0.34 


0.48 




0.64 


0.32 




0.73 


0.32 


0.61 


C 


1.79 


0.41 


C 


1.89 


0.41 


0.61 




-0.42 


0.40 




•0.42 


0.41 
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TASLE 3 

SInQlt, CoMblntdf and Avtragtd DIP StttUtlct 
Critorlon r Tottl Tttt Scort 



fOW 7 FORM 11 



ITEM 


CLASS 


NN-DDIF 


SE 


ITEM 


CLASS 


MTfOOIl 


3 


8 


•1.48 


0.52 


43 




1.03 


4 




•0.67 


0.53 


39 




•0.04 


5 




0.08 


0.44 


54 




0.66 


7 




0.02 


0.46 


64 




•C.24 


9 




•0.35 


0.S8 


33 




•C.60 


10 




0.71 


0.42 


82 




0.87 


1A 




0.94 


0.41 


1 


8 


1.58 


18 




o.r.2 


1.03 


2 




•1.52 


26 




•1.25 


1.50 


72 




•0.84 


28 




0.14 


0.39 


87 




0.61 


33 




•0.57 


0.53 


48 




•..04 


37 




0.27 


0.39 


92 




0.38 


38 


8 


1.65 


0.58 


16 


8 


1.80 


42 


8 


1.76 


0.80 


27 




1.16 


44 




0.81 


0.52 


44 




0.02 


53 




0.83 


C.40 


88 




•0.07 


60 




•0.40 


0.57 


28 




•0.54 


62 




•0.02 


0.41 


90 




•0.15 


67 




•0.74 


0.60 


31 




•0.22 


68 


8 


•1.13 


0.40 


89 




79 


75 


8 


1.21 


0.41 


91 


8 


tf.41 


81 




•0.19 


0.50 


38 




0.17 


83 




•0.09 


0.66 


67 




0.14 


85 


C 


2.76 


0.72 


29 




O.XO 


86 




•0.61 


0.38 


94 




•0.88 


88 




0.61 


0.70 


22 




0.84 


89 




•0.22 


0.44 


63 




0.44 


91 




0.93 


0.48 


61 




0.24 


92 




0.80 


0.43 


73 




0.86 


95 


c 


2.12 


0.56 


41 


8 


1.83 



HEAM ^0.26 0.56 -0.-3 



C0N8INE0 AVERAGE 



SE 


CLASS 


NH D^DIF 


SE 


CLASS NH D^DIF 


SE 


0.64 




•1.22 


0.40 


8 -1.28 


0.41 


0.60 




•0.54 


0.39 


•0.37 


0.40 


0.51 




0.42 


0.33 


0.35 


0.33 


0.58 




•0.19 


0.35 


•0.10 


0.36 


0.68 




•0.53 


0.43 


•0.47 


0.44 


0.49 




0.70 


0.31 


0.78 


0.32 


0.53 


8 


1.27 


T).32 


8 1.22 


0.33 


0.94 




•0.56 


0.67 


•0.55 


0.70 


1.42 




•1.08 


1.04 


•1.04 


1.03 


0.44 




0.29 


0.29 


0.36 


0.29 


0.62 




•0.69 


0.40 


•0.79 


0.40 


0.45 




0.35 


0.29 


0.32 


0.30 


0.67 


8 


1.64 


0.43 


8 1.72 


0.44 


0.83 


8 


1.29 


0.55 


8 1.47 


0.58 


0.65 




0.48 


0.40 


0.46 


0.41 


0.47 




0.47 


0.30 


0.42 


0.31 


0.68 




•n ^ 

U.90 




•U.46 


0.44 


0.48 




•0.16 


0.31 


0.08 


0.31 


0.70 




•0.64 


0.45 


•0.50 


0.46 


0.46 


8 


•1.07 


0.30 


•0.97 


0.30 


0.48 


8 


1.26 


0.31 


8 1.30 


0.31 


0.63 




•0.15 


0.39 


•0.03 


0.39 


0.97 




0.07 


0.54 


0.00 


0.56 


0.74 


8 


1.74 


0.49 


8 1.55 


0.52 


0.46 




•0.65 


0.29 


•0.73 


o.L-* 


0.81 




0.03 


0.52 


0.72 


0.53 


0.52 




0.24 


0.33 


0.06 


0.34 


0.50 




0.69 


0.34 


0.59 


0.35 


0.50 




0.85 


0.32 


0.83 


0.33 


0.62 


t 


1.94 


0.41 


C 1.98 


0.42 


0.64 




•0.27 


0.41 


•0.25 


0.42 



ERIC 



IG 



TAILE 4 

Sin^lt, ConbifMd, ind Avtragtd DIP Statistics 
Crittrioh « CoMson Itsns Scort 



ITEM 


CLASS 


NK-DDIP 


u 


2 




•0.01 


0.52 


4 




-0.60 


0.47 


7 




0.00 


0.55 


8 




0.50 


o.a 


11 




0.19 


0.49 


15 




0.85 


0.61 


16 




•0.52 


0.51 


17 




0.32 


0.51 


18 




•0.28 


0.58 


24 




•0.48 


0.46 


27 




•0.32 


0.50 


30 




•0.61 


0.53 


32 




1.32 


0.70 


38 


1 


•1.09 


0.49 


40 




•0.19 


0.61 


44 




•1.40 


0.69 


45 


8 


1.60 


0.67 


/ 




1.19 


1.11 


49 




0.34 


0.48 


50 


8 


1.40 


0.55 


51 




•0.43 


0.82 


52 


8 


•1.20 


0.46 


53 




•0.57 


0.59 


54 




0.74 


0.45 


55 




0.21 


0.48 


58 




0.34 


0.49 


59 




•0.39 


0.60 


61 




•0.51 


0.90 


64 




0.38 


0.44 


66 

t 




•0.21 


0.52 


68 




0.20 


0.51 


70 


i 


1.97 


0.85 


71 


8 


1.57 


0.66 


72 




•0.89 


0.60 


74 


• 


•i.a 


0.51 


75 




0.01 


0.57 


76 


C 


3.85 


1.18 


78 




•0.51 


0.54 


79 


8 


•1.03 


0.48 


80 




•0.45 


0.46 


U 




0.54 


0.57 


85 


• 


1.08 


0.49 


86 




0.68 


0.53 


87 




•0.37 


0.50 


89 




•0.33 


0.46 


92 




0.30 


0.46 


93 




0.50 


0.43 


94 




0.11 


0.50 



mm '0.29 0.57 



er|c 



P08N 6 

ITEN CUSS NK D*DIP 8E CUSS 



63 •0.61 0.47 

91 •0.80 0.46 
14 8 1.93 0.63 
48 0.16 0.42 

7 0.21 0.48 

80 0.77 0.58 
71 ^0.59 0.%8 

5 O.a 0.46 
4 0.24 0.57 

66 -0.36 0.43 

6 0.11 O.a 
12 0.82 0.53 

9 1.09 0.60 8 

2 8 -1.17 D.a 8 

26 1.33 0.70 

78 ^0.85 0.60 8 

25 8 1.46 0.60 8 

87 1.66 0.89 B 

69 0.04 O.a 

19 0.16 0.50 

29 -0.55 0.75 

30 •0.49 0.43 

84 -0.20 0.46 

85 0.1S 0.42 

81 -0.27 0.45 

33 8 1.09 0.49 

34 -0.29 0.58 

20 0.27 0.71 
36 -0.19 0.43 

86 -0.25 O.a 
90 -0.29 0.50 

92 0.63 0.65 i 

54 1.10 0.56 i 
W .1.39 0.62 i 

70 - 0.67 0.49 i 
57 0.13 0.52 

*7 0.05 0.77 8 

67 -0.21 0.50 
49 .0.10 O.a 
62 .0.41 0.43 
59 0.00 0.48 
81 0.07 0.49 
15 ^ 0.38 0.60 
53 .0.17 0.45 
85 0.35 0.43 

55 -0.51 O.a 
8 0.82 0.43 

52 0.76 0.46 



•0.23 0.52 

17 



C0N8INED AVERAGE 



NK D*DIP 


SE 


CUSS 


NK D'DIP 


SE 


•0.37 


0.34 




•0.33 


0.35 


•0.81 


0.33 




•0.70 


0.33 


0.89 


0.40 




0.90 


0.42 


0.31 


0.30 




0.33 


0.30 


0.22 


0.34 




0.20 


0.34 


0.77 


0.41 




0.81 


0.42 


•0.56 


0.34 




•0.56 


0.35 


0.29 


0.33 




0.38 


0.34 


•0.06 


0.40 




•0.02 


0.41 


•0.45 


0.31 




•0.42 


0.31 


•0.07 


0.33 




•0.09 


0.33 


0.13 


0.37 




0.10 


0.37 


1.10 


0.45 


8 


1.20 


0.46 


•1.11 


0.32 


8 


•1.13 


0.33 


0.43 


0.45 




0.52 


0.46 


•1.08 


O.a 


8 


•1.11 


0.45 


1.61 


o.a 


B 


1.53 


0.45 


1.46 


0.68 


B 


1.45 


0.70 


0.12 


0.32 




0.18 


0.32 


0.60 


0.36 




0.75 


0.37 


•0.56 


0.55 




•0.49 


0.55 


•0.80 


0.30 




•0.83 


0.31 


•0.33 


0.36 




•0.36 


0.37 


O.a 


0.30 




0.43 


» 0.31 


0.00 


0.32 




•0.04 


0.33 


0.74 


0.35 




0.72 


0.35 


•0.36 


0.42 




•0.34 


0.42 


0.12 


0.55 




•0.07 


0.56 


•0.01 


0.30 




0.09 


0.31 


•0.11 


0.33 




•0.23 


0.34 


•0.05 


0.35 




•0.05 


0.36 


1.06 


0.50 


B 


1.21 


0.52 


1.38 


0.42 


B 


1.32 


0.43 


•1.08 


0.42 


B 


•1.14 


0.43 


•1.20 


0.35 


B 


•1.05 


0.35 


0.07 


0.38 




0.07 


0.38 


1.28 


0.61 


8 


1.55 


0.6i« 


•0.31 


0.36 




•0.35 


0.37 


•0.58 


0 32 




.n 

•U.3% 




•0.43 


0.31 




•0.43 


0.31 


0.21 


0.36 




0.25 


0.37 


0.53 


0.34 




0.51 


0.35 


0.18 


0.39 




0.18 


0.40 


•0.13 


0.33 




•0.26 


0.33 


0.10 


0.31 




0.02 


0.31 


•0.14 


0.32 




•0.11 


0.32 


0.59 


0.30 




0.66 


0.30 


•0.32 


0.33 




0.45 


0.34 


•0.23 


0.38 




•0.22 


0.39 



TAILE S 

Singlt, Coiblntd, and Avtragtd DIF StatUtfcs 
CrlttrlorTs Total Ttst Scort 



FOKM 4 
ITEM CLASS NK*ODIF 



9 
C 




-0.06 


* 




-0.66 


f 




-0*05 


m 

O 




0«S6 


n 




0*19 


19 




0*87 


10 




-0.26 


If 




A 

0«18 


10 




A ^1 

-0*42 


9Z 
£h 




-0.55 


9y 

cr 




- A 49 

•0.17 






A <1A 

-0*20 




i 


1*38 


9Q 




•0«y6 


40 




•0.39 






-1.46 


45 


1 


1.75 


46 




1.67 


4y 




0.30 


EA 


1 


1.11 


51 




•0.05 


52 


1 


•1.00 


53 




•0.47 


54 




0.81 


KC 

33 




0.46 


30 




'^.26 


KO 

sv 




•0.30 


01 




0.13 


04 




0.25 


00 




•0.03 


Aft 

00 




0.06 


rU 


M 

9 


1.46 


y\ 


i 


1.55 


73 
fc 




A 

•0.74 




■ 


•1.74 


f 3 




A AX 

0.06 


rO 


c 


3.00 


78 




•0.55 


TV 


1 


•1.03 


80 




•0.29 


84 




0.41 


85 


1 


1.57 


86 




0.62 


87 




0.05 


89 




•0.30 


92 




0.50 


93 




0.30 


94 




0.17 



MEAN -0.24 

er|c 



FORN 6 



COMBINED 



AVERAGE 



SE 


ITEM 


CLASS MD-DOIF 


SE 


CLASS MK D^DIF 


SE 


0.52 


63 


•0.65 


0.48 


•0.25 


0.48 


0.47 


91 


•0.86 


0.47 


•0.75 


0.33 


A KX 

0.56 


14 


B 2.06 


0.68 


0.79 


0.41 


0.44 


48 


0.14 


0.43 


0.23 


0.30 


0.49 


7 


0.04 


0.49 


0.23 


0.34 


0.63 


80 


0.50 


0.60 


0.62 


0.42 


0.52 


71 


•0.53 


0.48 


•0.42 


0.35 


0.51 


5 


0.55 


0.47 


0.40 


0.33 


0.58 


4 


0.23 


0.59 


•0.01 


0.40 


A /X 

0.46 


66 


•0.50 


0.43 


•0.48 


0.31 


A XA 

0.4y 


6 


0.14 


0.45 


0.01 


0.32 


A C/ 

0.54 


12 


0.71 


0.54 


0.22 


0.37 


A XA 

D.Oy 


9 


B 1.18 


0.62 


B 1.24 


0.45 


A / • 

0.48 


2 


B •I. 20 


0.45 


B -1.11 


0.32 


A X9 

0.62 


26 


1.31 


0.70 


0.35 


0.44 


0.69 


78 


•0.60 


0.63 


B -1.00 


0.45 


0*67 


25 


B 1.32 


0.64 


B 1.55 


0.44 


1.07 


87 


1.94 


1.06 


B 1.83 


0.74 


0.48 


69 


•0.03 


0.45 


0.18 


0.32 


0.53 


19 


0.04 


0.50 


0.58 


0.36 


0«81 


29 


•0.52 


0.75 


•0.31 


0.54 


0.46 


30 


•0.47 


0.43 


•0.75 


0.31 


0.48 


84 


•0.20 


0.47 


•0.27 


0.36 


0.46 


85 


0.07 


0.43 


0.31 


0.30 


0.48 


81 


•0.22 


0.46 


0.00 


0.32 


A 4A 

0«1B 


33 


0.98 


0.50 


0.62 


0.34 


D«12 


34 


•0.18 


0.61 


•0.30 


0.42 


0.87 


20 


0.34 


0.74 


0.36 


0.55 


0.22 


36 


•0.19 


0.44 


•0.06 


0.31 


A K V 

0.53 


86 


•0.24 


0.45 


•0.10 


0.33 


A K 4 

0«51 


90 


•O.SO 


0.50 


•0.14 


0.35 


A 


92 


0.31 


0.69 


0.V2 


0.51 


A XK 


54 


B 1.45 


0.61 


B 1.38 


0.43 


A KA 


89 


•1.15 


0.62 


•0.89 


0.42 


D«54 


9A 

70 


•0.57 


0.51 


B -1.04 


0.35 


A KA 

0.58 


57 


0.14 


0.54 


0.07 


0.38 


1 9C 


ty 


0.31 


0.83 


B 1.37 


0.63 




Or 


0.00 


0.51 


•0.35 


0.37 


0.48 


49 


•0.27 


0.45 


•0.59 


0.32 


0.45 


62 


•0.21 


0.43 


•0.33 


0.31 


0 S7 

w.^ r 




A AX 


A ^ A 


0.27 


0.36 


0.51 


61 


0.01 


0.49 


0.66 


0.34 


0.54 


15 


•0.58 


0.62 


0.10 


0.40 


0.50 


53 


•0.04 


0.46 


•0.05 


0.33 


0.46 


65 


0.51 


0.44 


0.14 


0.31 


0.48 


55 


•0.29 


0.46 


•0.06 


0.32 


0.44 


8 


0.55 


0.43 


0.45 


0.30 


0.50 


52 


•0.86 


0.48 


•0.28 


0.34 


0.56 




•0>23 


0.54 


•0.20 


0.38 



CLASS NK D-LiF SE 



•0.37 
•0.76 
0.90 
0.25 
0.12 
0.68 
•0.40 
0.37 
•0.10 
•0.52 
•0.01 
0.26 
1.27 
•1.08 
0.41 
•1.01 
1.53 
1.81 
0.13 
0.56 
-0.29 
•0.73 
•0.33 
0.43 
0.11 
0.45 
•0.28 
0.24 
0.10 
•0.14 
•0.22 
0.84 
1.50 
•0.94 
•1.14 
0.10 
1.56 
•0.22 
•0.64 
•0.25 
0.22 
0.77 
0.06 
0.00 
0.11 
0.10 
0.43 
•0.36 



0.35 

0.33 

0.43 

0.31 

0.35 

0.43 

0.35 

0.35 

0.41 

0.31 

0.33 

0.38 

0.46 

0.33 

0.46 

0.47 

0.46 

0.75 

0.33 

0.36 

0.55 

0.31 

0.34 

0.31 

0.33 

0.19 

0.14 

0.57 

0.21 

0.34 

0.36 

0.53 

0.45 

0.43 

0.37 

0.40 

0.71 

0.43 

0.33 

0.31 

0.37 

0.35 

0.41 

0.34 

0.32 

0.33 

0.31 

0.35 



•0.20 0.38 



IS 



TAILE 6 

Sinolt, CoMblntd, and Avtragtd OIF Statistics 
Critarion • Coamon Itaaia Score 



FORN 3 PORN 5 



ITEN 


CLASS 


NH-DOIF 


$E 


ITEN 


CLASS 


M>^-DDIP 


3 




0.38 


0.48 


94 




0.21 


7 




•0.15 


0.57 


28 




0.07 


9 




0.31 


0.38 


49 




0.32 


12 




-0.26 


0.42 


39 




0.57 


13 




0.29 


0.57 


68 




•1.04 


H 




•0.33 


0.39 


59 


8 


•1.40 


21 




0.30 


0.42 


35 




0.42 


2S 




0.2S 


0.42 


51 




•0.14 


26 




•0.25 


0.49 


56 




'0.64 


28 




0.38 


0.48 


31 




•0.35 


29 




*0.12 


0.54 


2 




•0.39 


30 




*0.41 


0.40 


18 




*0.30 


31 




•0.16 


0.41 


90 




0.63 


32 




0.28 


0.39 


45 




•0.12 


34 




0.07 


0.45 


86 




•0.86 


35 


8 


•1.46 


0.59 


82 




•0.51 


37 




-0.28 


0.39 


8 




0.16 


39 




-0.18 


0.41 


70 




0.32 


40 




0.04 


0.43 


95 




0.22 


45 




0.94 


0.53 


85 


1 


1.46 


46 




•0.15 


0.39 


61 




•0.27 


48 




0.90 


0.55 


73 




0.07 


49 




0.40 


0.40 


69 


B 


1.02 


51 




-0.22 


0.41 


67 




•0.09 


52 


B 


•1.01 


0.43 


23 


B 


•1.74 


54 




•1.37 


0.83 


7 




0.25 


60 




0.22 


0.61 


58 




1.27 


62 




0.12 


0.42 


14 




0.82 


64 


8 


•1.22 


'0.60 


25 




•0.72 


65 




0.37 


0.48 


75 




0.30 


68 




0.10 


0.43 


66 




0.07 


69 




0.52 


0.47 


48 




0.80 


70 




•0.62 


0.40 


20 


B 


•1.36 


72 




0.17 


0.38 


42 




0.47 


73 


8 


2.45 - 


1.15 


22 




1.73 


74 




0.31 


0.41 


76 


B 


1.04 


75 


8 


1.15 


0.58 


79 


B 


1.89 


77 




0.66 


0.40 


37 




•0.29 


78 




*0.35 


0.37 


5 




•0.08 


80 




•1.14 


0.84 


83 




•i.a 


81 




1.10 


0.82 


36 




0.65 


83 




•0.29 


0.45 


3 




0.80 


84 




0.39 


0.38 


30 




•0.28 


85 




•0.03 


1.36 


32 




•0.38 


86 




0.52 


0.62 


92 




•0.49 


87 




0.76 


0.39 


27 




0.66 


88 




•1.23 


0.71 


87 




•0.27 


89 




0.45 


0.50 


44 




0.24 


91 




•0.45 


0.45 


24 




•0.26 


92 




•0.26 


0.63 


1 




0.27 


93 




•0.61 


0.67 


38 


B 


•1.44 


94 




•0.93 


0.39 


84 




•0.89 


MEAN 




•0.26 


0.52 






•0.30 



er|c 



COMBINEO AVERAGE 



SE 


CLASS 


NH 0^01 F 


SE 


CLASS MH 0-OlF 




0.53 




0.38 


0.35 


0.30 


0 16 

V.JO 


0.58 




0.75 


0.28 


•0.04 


0.41 


0.43 




0.37 


0.28 


0.31 


0.29 


0.44 




0.13 


0.30 


0.15 


0 If) 

U. JW 


0.62 




•0.26 


0.41 


•0.35 


0.42 


0.42 




•0.86 


0.28 


•0.85 


0 90 

U.£T 


0.45 




0*39 


0.31 


0 36 

W. JV 


0 31 

U.J 1 


0.42 




0.01 


0.29 


0.05 


0 10 

U. JU 


0.50 




•0.44 


0.35 


• 0 LL 


0.33 


0.50 




0.05 


0.34 


0.02 


0 IS 

U. J^ 


0.51 




•0.34 


0.37 


•0.26 


0 37 

U. Jf 


0.43 




•0.38 


0.29 


*0 lA 

U.JO 


0 29 

U.CT 


0.43 




0.25 


0.30 


0 7% 


0 10 

U. JU 


0.41 




0.08 


0.28 


0.09 


0 98 
u.co 


0.46 




•0.38 


0 32 


10 

l^. JT 


0 19 

U. Jft 


0.57 


B 


•1.01 


0.41 


•0 08 

W .TO 


0.41 


0.42 




•0.13 


0.28 


•0.07 


0 90 

U.£T 


0.44 




0.04 


0.30 


0 0^ 

V .uo 


0 10 
U. JU 


0.45 




0.11 


0 31 


0 11 
U. 1 J 


0 11 

U. j1 


0.56 


B 


1.18 


W.JO 


B 1 10 

■ 1 . IT 


0 10 
u. jy 


0.43 




•0.18 




U.C 1 


0 90 


0.74 




0.50 


0 ^3 
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